Quality Control in Crowdsourcing: An Objective Measurement Approach to Identifying and Correcting Rater Effects in the Social Evaluation of Products and Services

Crowdsourcing requires new strategies to evaluate the workers involved as well as the . Using customer feedback data, we introduce multi-facetted Rasch scaling as an evaluation technique to assess the contributions of workers and products simultaneously within a single coherent measurement framework. Based on a data set of about 250,000 customers who rated nearly 115,000 products, for a total of nearly 3 million cases, we found that product ratings reflect almost as much the existence of stable rater differences as they are We illustrate how Rasch scaling provides extensive quality control mechanisms; as well we show how aberrant workers and products can be identified so that appropriate feedback and/or corrective actions can be initiated.